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Figure  1 .  Conceptualization  of  the  complex  aviation  environment  of  the  21st  century. 

Cognitive  Systems  Engineering  in 
Military  Aviation  Domains: 

An  Introductory  Primer 


Michael  D.  McNeese  and 
Michael  A.  Vidulich 


It  is  our  privilege  to  welcome  you  to  a  special 
issue  of  the  HSIAC  Gateway  that  introduces 
readers  to  cognitive  systems  engineering  (CSE) 
as  it  relates  and  applies  to  military  aviation 
domains.  In  this  issue  we  provide  a  broad  sam¬ 
pling  of  perspectives,  issues,  methods,  and  appli¬ 
cations  that  afford  a  first-look  understanding  of 
CSE  for  use  within  aviation  fields  of  practice. 
There  will  be  nine  separate  synopses  provided  to 
whet  your  interest  in  the  forthcoming  HSIAC 
state-of-the-art  report  (SOAR),  Cognitive  Systems 
Engineering  in  Military,  Aviation  Domains: 
Avoiding  Cogminutia  Fragmentosa!  (McNeese  & 
Vidulich,  Eds.,  2001).  Each  article  summarizes  a 
chapter  to  appear  in  the  report.  Now  take  a  clos¬ 
er  look  at  what  one  may  expect  for  the  special 
issue — and  in  turn  the  SOAR. 


Complex  environments  of  the  21st  cen¬ 
tury  place  workers  in  an  information-rich 
world  (see  Figure  1)  with  little  time  to 
make  sense  out  of  events  surrounding 
them,  assess  their  plans,  make  appropri¬ 
ate  decisions,  or  perform  multiple  activi¬ 
ties.  In  many  cases,  computational  sup¬ 
port  and  advanced  interfaces  for  work 
activities  have  not  been  engineered  with 
cognition  or  context  in  mind. 
Unfortunately,  this  lack  of  “cognitive 
engineering”  may  produce  what  we  refer 
to  as  cogminutia  fragmentosa,  where  the 
worker’s  cognitive  world  breaks  down 
into  small,  isolated  strands  of  thought  as 
unanticipated  events  transpire  (mental 
stovepipes) .  There  can  be  a  loss  of  mean- 
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ing  or  control  as  the  worker  becomes  sep¬ 
arated  from  the  demands  of  his  or  her 
work  and  may  remain  lost  in  terms  of 
comprehending  the  emerging  elements  of 
a  situation.  When  cogminutia  fragmen- 
tosa  persists,  there  is  no  longer  an  inter¬ 
face  between  the  worker’s  cognitive 
world  and  the  work  for  which  he  or  she 
is  responsible.  In  other  words,  the  work¬ 
er  cannot  properly  adapt  to  the  situation 
encountered  (i.e.,  a  maladaptive  state 
exists) .  If  this  state  continues,  errors,  fail¬ 
ure,  and  even  catastrophic  disasters  are 
highly  probable.  This  state  may  also  con¬ 
tribute  to  affective  and  emotional 
responses  (e.g.,  fear,  anxiety,  rage)  that 
further  complicate  agent-environment 
transactions.  However  all  is  not  lost.  We 
are  now  at  a  point  in  history  where  it  is 
not  uncommon  to  observe  human  factors 
practitioners  referring  to  “cognitive  sys¬ 
tems  engineering”  as  their  method  or  tool 
of  choice  to  respond  to  work  environ¬ 
ments  that  produce  cogminutia  fragmen- 
tosa.  Indeed,  as  first-of-a-kind  cognitive 
systems  are  proposed  for  complex  envi¬ 
ronments,  such  as  in  military  aviation 
domains,  CSE  is  frequently  utilized  to 
understand  and  analyze  various  compo¬ 
nents  of  operator  or  team  expertise  (e.g., 
cognitive  skills,  engagement  rules,  specif¬ 
ic  knowledge),  and  the  interaction  of 
expertise  with  specifications  of  the  work 
domain.  As  CSE  is  applied  to  real-world 
settings,  agent-environment  transactions 
can  be  quantitatively  or  qualitatively 
modeled  (represented)  and  then  used  as 
a  basis  to  predicate  elements  of  a  design 
(e.g.,  a  human-computer  interface  or  a 
decision-support  system).  Typically,  CSE 
practitioners  engage  workers  through  a 
variety  of  CSE  methods  that  capture  mul¬ 
tiple  facets  of  how  work  is  transacted 
from  agents  to  environment. 

This  special  issue  highlights  the  per¬ 
spectives  and  foundations  of  an  interna¬ 
tional  community  of  practitioners  who 
have  both  developed  and  applied  CSE. 
One  can  see  the  field  emerges  from  sev¬ 
eral  corridors  that  in  turn  produce  alter¬ 
native  methodologies/approaches  to 
address  military  aviation  domains. 
Differing  philosophies  and  techniques 
spawn  incisive  pathways  of  integration 
in  the  development  of  design  artifacts. 
Because  the  aviation  domain  is  fraught 
with  multifarious  levels  of  complexity 
and  is  demonstrative  of  cogminutia 


fragmentosa,  we  believe  it  supplies  an  excellent 
foundation  for  reviewing,  assessing,  communicat¬ 
ing,  and  evaluating  some  of  the  principles  (and 
nuances)  inherent  within  various  programs  of 
CSE.  The  SOAR  will  emulate  this  objective  by  pre¬ 
senting  nine  chapters  in  the  following  three  sec¬ 
tions  for  readers  (along  with  the  respective  first 
author  of  each  chapter) : 

•  Foundations  and  Perspectives  (Reising, 
Eggleston,  McNeese,  Woods) 

•  Methodological  Pursuits  (Roth,  Naikar,  Hendy) 

•  Innovations,  Integration,  and  Application 
(Taylor,  Hudlicka) 

The  forthcoming  synopses  briefly  traverse  these 
sections  and  broadly  define  the  waypoints  of  the 
SOAR,  hence  providing  readers  with  an  informed 
introduction  to  our  special  issue  topic.  They  chal¬ 
lenge  the  reader  to  contrast/compare  philosophies 
of  use,  theories  of  origin,  goals,  benefits,  methods, 
tools,  experiences,  constraints  and  problems  of 
applications,  lessons  learned,  and  examples  as  a 
means  to  generate  new  levels  of  understanding — 
as  they  relate  to  the  specific  constraints  encoun¬ 
tered  in  military  aviation. 
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Cognitive  Engineering  and 
its  Relationship  to 

Future  Aviation  Systems 

John  M.  Reising 


John  Reising,  Ph.D.,  is  with 
the  Crew  System  Interface 
Division,  Human  Effectiveness 
Directorate,  U.S.  Air  Force 
Research  Laboratory,  Wright- 
Patterson  Air  Force  Base, 
Ohio. 


The  theme  of  cognitive  systems 
engineering  (CSE)  presupposes 
two  things:  first,  that  military 
aviation  systems  are  susceptible  to  cog- 
minutia  fragmentosa;  second,  that  the 
application  of  cognitive  systems  engi¬ 
neering  to  the  design  of  military  avia¬ 
tion  systems  is  a  likely  palliative  for  the 
problem.  Each  of  these  presuppositions 
requires  independent  consideration. 
The  focus  here  is  on  the  viability  of  the 
assumption  that  the  design  of  military 
aviation  systems  could  promote  cog- 
minutia  fragmentosa.  In  considering 
this  issue,  it  will  be  suggested  that  not 
only  is  cogminutia  fragmentosa  a  risk  in 
current  military  aviation  systems,  but 
that  the  technological  trends  are  making 
it  an  increasing  risk  in  the  future. 

One  of  the  hallmarks  of  cogminutia 
fragmentosa  is  that  the  user  of  a  system 
cannot  effectively  adapt  to  encountered 
situations.  This  inability  is  due  to  a  mis¬ 
match  between  the  human’s  under¬ 
standing  of  the  current  situation’s  prop¬ 
erties,  potentiality,  and  constraints  and 
the  actual  state  of  the  real  world.  Such 
mismatches  are  certainly  present  in 
today’s  military  aviation  systems. 

One  of  the  best  examples  of  the  prob¬ 
lem  is  the  difficulty  that  has  been  expe¬ 
rienced  in  effectively  using  automation 
to  aid  pilots.  A  common  problem  with 
the  implementation  of  aviation  automa¬ 
tion  has  been  the  moving  of  the  pilot 
further  and  further  out  of  the  aircraft’s 
direct  control  loop.  In  some  cases  the 
pilot  became  a  relatively  inactive  part¬ 
ner  of  the  automated  system  in  the  con¬ 
trol  of  the  aircraft.  All  often  appeared  to 
be  fine,  until  something  unexpected 
occurred.  The  automated  systems  could 
not  be  designed  to  deal  with  unexpect¬ 
ed  situations  and  the  “out-of-the-loop” 
pilot  was  suddenly  confronted  with  a 
bad  situation  with  little  or  no  knowl¬ 


edge  of  how  it  came  to  occur.  Although  this  prob¬ 
lem  has  been  under  consideration  since  World  War 
II,  there  has  not  been  any  definitive  solution  found 
for  even  the  current  generation  of  relatively  simple 
automated  aids. 

Previously,  automation  was  largely  used  to  con¬ 
trol  little  more  than  the  aircraft’s  flight  path. 
Currently,  automation  is  taking  over  many  more 
tasks  in  the  cockpits  of  such  aircraft  as  the  Royal 
Air  Force’s  GR-1  Tornado  and  the  U.S.  Air  Force’s 
F-15E  Strike  Eagle.  This  change  in  the  role  of 
automation  from  simple  aids  for  specific  tasks  to 
complicated  collections  of  automated  aids  that 
participate  in  a  multitude  of  the  pilot’s  tasks,  has 
effectively  made  automation  a  part  of  the  aircraft’s 
crew.  Problems  occur  because  the  pilots  and  flight 
crewmembers  often  do  not  understand  what  the 
automation  is  doing  or  why.  Much  precious  time 
and  effort  must  be  expended  in  analyzing  the 
actions  of  the  automation,  rather  than  in  adapting 
to  the  current  mission  situation.  In  other  words, 
automation  is  often  a  bad  team  member  that  pro¬ 
motes  cogminutia  fragmentosa. 

To  improve  this  situation,  automation  must 
become  more  sophisticated  in  its  interactions  with 
the  human  team  members.  This  more  sophisticat¬ 
ed  and  cooperative  future  automation  has  been 
referred  to  as  an  “electronic  associate”  (EA). 
Designing  the  EA  demands  a  much  better  under¬ 
standing  of  the  human  crew  so  that  the  EA  can 
work  well  with  the  crew.  Cognitive  systems  engi¬ 
neering  may  provide  the  insights  to  accomplish 
this  goal.  It  is  vital  that  useful  tools  for  avoiding 
cogminutia  fragmentosa  be  developed  and  validat¬ 
ed  because  pilots  of  future  military  aircraft  (such 
as  the  F-22)  will  be  confronted  with  even  more 
capable  and  complex  systems  to  control.  Also, 
some  future  systems,  such  as  the  uninhabited 
combat  aerial  vehicle  (UCAV)  (see  Figure  1),  will 
be  moving  the  operator  completely  out  of  the  air¬ 
craft.  The  operators  of  such  systems  need  the 
automation  controlling  the  systems  to  be  a 
dependable  partner.  Using  cognitive  systems  engi¬ 
neering  should  be  tested  as  a  means  for  designing 
such  challenging  and  important  future  systems.* 
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Figure  1.  Adaptive  UCAV  system  diagram. 


HSIAC  Pulse  Check  Reflected  by  Web  Site 


The  HSIAC  Web  site,  hosted  by  the  Defense  Technical 
Information  Center  (DTIC),  at  http://iac.dtic.mil/hsiac. 
is  growing  and  expanding  in  scope.  Changes  to  this 
site  have  been  dramatic  and  reflect  the  activity  going  on  at 
the  Human  Systems  I  AC. 

The  major  driving  force  behind  the  redesign  is  the  expand¬ 
ed  scope  of  the  IAC’s  subject  areas,  which  are  called 
Domains.  Our  charter  now  includes  eleven  domains  of 
expertise  including: 

1.  Human  Systems  Integration, 

2.  Habitability, 

3.  Human  Factors, 

4.  Safety, 

5.  Health  Hazards, 

6.  Medical  Factors, 

7.  Personnel  Survivability, 

8.  Manpower, 

9.  Personnel, 

10.  Training, 


and  the  previous  three  (8,  9,  10)  combined  as  11,  MPT. 

The  Human  Systems  IAC  has  specific  areas  of  specialized 
knowledge  referred  to  as  Pillars.  We  currently  sponsor  a 
Pillar  in  the  area  of  spatial  disorientation  (SD)  countermea¬ 
sures  by  working  with  the  Air  Force  Research  Laboratory  and 
Veridian  Engineering.  The  SD  pillar  can  be  accessed  from 
the  IAC  web  page  from  http://www.spatiald.wpafb.af.mil. 
This  growing  site  is  loaded  with  current  information  on  spa¬ 
tial  disorientation  and  will  display  the  results  of  ongoing  and 
developing  research  for  many  years  to  come. 

Other  changes  to  the  web  site  include  sites  where  free 
information,  such  as  specifications,  standards,  handbooks, 
magazines,  etc.  on  our  various  domains  can  be  obtained. 
This  information  is  provided  in  the  form  of  links. 
Summarized  HSIAC  product  information  and  a  section  called 
IAC  success  stories  highlighting  HSIAC' s  activities  will  soon 
be  available.* 
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The  State  of  Cognitive  Engineering: 
A  Retrospective-Based  Analysis 


Robert  G.  Eggleston 


Cognitive  systems  engineering 
(CSE)  is  a  multidisciplinary  field 
that  is  both  a  scientific  endeavor 
and  an  engineering  practice  that 
emphasizes  human-centered  analysis 
and  design.  I  have  recently  completed 
an  assessment  of  the  state  of  the  CSE 
field  by  way  of  an  informal  retrospec¬ 
tive  analysis.  This  analysis,  extending 
back  to  the  origins  of  CSE,  considers  the 
field  from  three  different  perspectives: 
conceptual  foundations,  engineering 
tools  and  practices,  and  the  state  of 
deployment.  Some  unique  features  of 
the  analysis  include: 


•  Depiction  and  contrast  of  several 
different  CSE  frameworks  motivat¬ 
ed  by  different  theoretical  positions 
and  applied  interests; 

•  Emphasis  on  both  theoretical  fac¬ 
tors  and  engineering  factors  of 
CSE;  and 

•  A  brief  look  at  issues  in  the 
deployment  of  CSE  across  the  full 
spectrum  of  analysis,  design,  and 
evaluation  components  of  develop¬ 
ment,  including  impediments  to 
progress  in  using  CSE  concepts, 
tools,  and  methods. 

This  article  is  a  brief  synopsis  of  the  ret¬ 
rospective  analysis  of  CSE. 

A  central  theoretical  focus  of  CSE  is  the 
study  of  work  representations  directly 
useful  for  design  engineering.  As  an  engi¬ 
neering  practice,  it  formulates  human- 
centered  work  analyses,  design  concepts, 
and  methods  to  facilitate  integration  of 
CSE  products  into  large-scale  system 
engineering  practices  (see  Figure  1). 

The  conceptual  foundations  of  the 
CSE  field  consist  of  different  frameworks 
for  modeling  work.  Four  CSE  frame¬ 
works  or  genotypes,  championed  by 
Card  (see  Eberts,  1997),  Norman  (1986), 


Figure  1.  The  field  of  Cognitive  Engineering  is  both  an 
engineering  practice  and  an  applied  science  in  the 
context  of  system  development. 

Rasmussen  (1988),  and  Woods  (see  Woods  &  Roth, 
1998),  serve  to  represent  the  scope  of  the  CSE  con¬ 
ceptual  landscape.  In  general,  these  frameworks 
place  different  emphasis  on  modeling  work  as  a 
property  of  (1)  an  agent,  (2)  a  task,  or  (3)  a  con¬ 
stellation  of  agent,  task,  and  environmental  factors. 

The  creation  of  these  CSE  genotypes  in  and  of 
themselves  indicates  progress  in  the  field.  Each  of 
them  provides  a  systematic  and  comprehensive 
conceptualization  of  how  to  model  work.  Some 
frameworks  follow  a  cognitivist  perspective, 
which  emphasizes  modeling  goal-directed  cogni¬ 
tive  work  in  information  processing  terms.  Others 
employ  an  ecological  perspective  that  regards 
cognitive  work  as  distributed  over  environmental 
and  agent  factors,  and  that  assigns  more  impor¬ 
tance  to  understanding  the  co-evolution  of  prob¬ 
lem  formation  in  work  and  ways  to  solve  the 
dynamically  evolving  work  problem. 

Currently,  the  state  of  the  various  genotypes  vary 
widely  in  terms  of  construct  specificity  and  deploy¬ 
ment  guidance.  In  general,  the  CSE  genotypes  orig¬ 
inating  with  Card  (cognitivist)  and  Rasmussen 
(ecological)  have  been  refined  to  a  greater  extent 
than  those  pioneered  by  Norman  and  Woods. 
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The  Field  of  Cognitive  Engineering 
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Figure  2.  Gains  and  limitations  of  cognitive  engineering  over  its  history  of  development. 


It  is  very  difficult  to  elicit  and  crisply  represent 
information  from  subject-matter  experts  that  cap¬ 
ture  the  cognitive  nature  and  demands  of  work. 
Considerable  attention  has  been  directed  toward 
the  development  and  refinement  of  new  cognitive 
task  analysis  methods  and  tools  to  improve  the 
CSE  field.  In  general,  many  new  probe  techniques 
have  been  developed  and  used  that  have  revealed 
new  insights  into  work.  These  range  from  tech¬ 
niques  that  focus  on  extracting  expertise  to  meth¬ 
ods  that  assess  different  forms  of  problem  solving 
and  decision  making  to  integrated  suites  of  tools 
that  attempt  to  provide  a  more  comprehensive  pic¬ 
ture  of  cognitive  work.  However,  to  date,  only  lim¬ 
ited  attention  has  been  directed  toward  the  relia¬ 
bility  of  tool  use  and  the  reusability  of  the  result¬ 
ant  outcome  products  over  a  system’s  life  cycle. 
Further,  the  gap  between  CSE-based  human-cen¬ 
tered  analysis  and  CSE  guides  to  innovative  artifact 
design  still  remains  large. 

CSE  concepts  and  methods  have  enjoyed  an 
increasingly  wide  use  in  system  engineering  in 
recent  years.  This  attests  to  the  perceived  value  of 
CSE.  But  on  the  negative  side,  perhaps  the  largest 
challenge  to  the  CSE  community  is  the  need  to 
improve  clarity  and  reduce  ambiguity  and  incon¬ 
sistencies  in  use  of  CSE  constructs  and  methods. 
Confusions  between  new  CSE  methods  and  older 
human  factor  methods  for  requirements  analysis 
abound.  Further,  technical  distinctions  across  CSE 
genotypes  do  not  always  appear  to  be  well  under¬ 
stood  and  followed,  both  within  and  between  dif¬ 
ferent  genotypes.  These  represent  serious  impedi¬ 
ments  to  the  continued  growth  of  the  field. 

In  sum,  while  CSE  has  made  a  great  deal  of  sci¬ 
entific  and  engineering  progress  over  twenty  some 
years,  there  are  areas  that  need  more  refinement 


before  CSE  is  likely  to  increase  its 
impact  on  system  development  (see 
Figure  2) .  ■ 
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Reno,  NV,  USA.  May  6-9,  2001. 

ASMA  2001:  Aerospace  Medical  Association  Meeting 

Contact  ASMA,  320  South  Henry  Street,  Alexandria,  VA  22314-3579,  USA. 
Tel:  +1-709-739-2240,  Fax:  +1-703-739-9652,  URL:  http://www.asma.ora 


Colorado  Springs,  CO,  USA.  May  14-17,  2001. 

46th  Biennial  Meeting  of  the  U.S.  Department  of  Defense  Human  Factors  Engineering  Technical 
Advisory  Group 

Contact  Sheryl  Cosing,  10822  Crippen  Vale  Court,  Reston,  VA  20194,  USA. 

Tel:  +1-703-925-9791,  Fax:  +1-703-925-9644,  E-mail:  sherylynn@aol.com, 

URL:  http://dticam.dtic.mil/hftaa/.  Meeting  is  open  to  all  government  personnel  and  others 
by  specific  invitation. 


San  Jose,  CA,  USA.  June  3-8,  2001 . 

SID  2001:  Society  for  Information  Display 

Contact  SID,  c/o  Palisades  Institute  for  Research  Services,  Inc.,  411  Lafayette  Street,  2nd 
Floor,  New  York,  NY  10003,  USA.  Tel :+1 -21 2-460-8090,  Fax:  +1-212-460-5460, 
E-mail:  wklein@palisades.org,  URL:  http://www.sid.org 


Fairfax,  VA,  USA.  June  4-7,  2001. 

XV  Annual  International  Occupational  Ergonomics  &  Safety  Conference 

Contact  Paul  Champney.  Tel:  +1-509-786-4689,  E-mail:  pchamp@quicktel.com, 
URL:  http://www.ISOES.org 


Miami  Beach,  FL,  USA.  June  4-7,  2001 . 

Intelligent  Transportation  Society  of  America’s  11,h  Annual  Meeting  and  Exposition 

Contact  ITS  America,  400  Virginia  Avenue,  SW,  Suite  800,  Washington,  DC  20024-2730. 
Tel:  +1-202-484-4847,  Fax:  +1-202-484-3483,  URL:  http://www.itsa.ora 


Arlington,  VA,  USA.  June  26-28,  2001. 

SAE  Digital  Human  Modeling  for  Design  and  Engineering  Conference  and  Exposition 

Contact  Paula  Preston,  SAE  International,  400  Commonwealth  Drive,  Warrendale,  PA 
15096-0001,  USA.  Tel:  +1-724-772-7131,  Fax:  +1-724-776-0002, 

E-mail:  meetings@sae.org,  URL:  http://www.sae.org/dhmc 
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Maui,  HI,  USA.  July  28-August  2,  2001. 

CAES  2001:  International  Conference  on  Computer-Aided  Ergonomics  and  Safety 

Contact  Waldemar  Karwowski,  Center  for  Industrial  Ergonomics,  Academic  Building, 
Room  445,  University  of  Louisville,  Louisville,  KY  40292.  Tel:  +1-502-852-7173, 
Fax:  +1-502-852-7397,  E-mail:  karwowski@louisville.edu, 

URL:  http://www.ergonet.net/caes2001  .html 


New  Orleans,  LA,  USA.  August  5-10,  2001. 

HCI  International  2001:  9th  International  Conference  on  Human-Computer  Interaction 

Contact  Kim  Gilbert,  School  of  Industrial  Engineering,  Purdue  University,  1287  Grissom 
Hall,  West  Lafayette,  IN  47907-1287,  USA.  Tel:  +1-765-494-5426, 

Fax:  +1-765-494-0874,  URL:  http://hcii2001  .engr.wisc.edu 


Nashville,  TN,  USA.  September  17-19,  2001. 

2001  SAFE  Annual  Symposium 

Contact  SAFE  Association,  107  Music  City  Circle,  Suite  112,  Nashville,  TN  37214,  USA. 
Tel:  +1-615-902-0056,  Fax  +1-615-902-0077,  E-mail:  safe@usit.net, 

URL:  http://safeassociation.org/2001  .htm 


Kassel,  Germany.  September  18-20,  2001. 

8lh  IFAC/IFIP/IFORS/IEA  Symposium  on  Analysis,  Design,  and  Evaluation  of  Human-Machine 
Systems  (HMS  2001) 

Contact  Gunnar  Johannsen.  E-mail:  hms2001@imat.maschinebau.uni-kassel.de, 
URL:  http://www.imat.maschinenbau.uni-kassel.de/hms2001/time2.html 


Minneapolis,  MN,  USA.  October  8-12,  2001. 

45th  Annual  Meeting  of  the  Human  Factors  and  Ergonomics  Society 

Contact  HFES,  PO  Box  1369,  Santa  Monica,  CA  90406,  USA.  Tel:  +1-310-394-1811, 
Fax:  +1-310-394-9793,  E-mail:  hfes@compuserve.com, 

URL:  http://hfes.oro.  Proposals  due  March  19,  2001 


Monterey,  CA,  USA.  November  2001 . 

47"’  Biennial  Meeting  of  the  U.S.  Department  of  Defense  Human  Factors  Engineering  Technical 
Advisory  Group 

Contact  Sheryl  Cosing,  10822  Crippen  Vale  Court,  Reston,  VA  20194,  USA. 

Tel:  +1-703-925-9791,  Fax:  +1-703-925-9644,  E-mail:  sherylynn@aol.com, 

URL:  http://dticam.dtic.mil/hftag/.  Meeting  is  open  to  all  government  personnel  and  others 
by  specific  invitation. 
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Discovering  How  Cognitive  Systems  Should  be 
Engineered  for  Aviation  Domains: 

A  Developmental  Look  at  Work, 

Research,  and  Practice 

Michael  D.  McNeese 


Table  1 .  Categorical  Structure  of  the  Book  Chapter. 


Considering  15  Years  Worth  of  What 

1. 

What  is  it? 

Defining  Characteristics/Core  Values 

II. 

What  are  the  formative  conditions? 

Background/History/Perspective 

III. 

What  are  the  objects  of  interest? 

Use/Directions/Application 

IV. 

What  are  representative  approaches/examples? 

Theories/Methods/Approaches 

V. 

What  has  transpired? 

Progress-Influences/Developing  Stages 

VI. 

What  has  evolved/What  has  been  learned? 

Lessons/  Learned/Recombinate  Themes 

VII. 

What  is  next? 

Emerging  Issues/Future  Directions 

As  the  old  adage  goes,  “necessity 
is  the  mother  of  invention.”  A 
15-year  retrospective  review  of 
the  author’s  work  at  Wright-Patterson 
Air  Force  Base  provides  the  basis  for 
discovering  insights  among  the  joint 
confluence  of  cognitive  science,  cogni¬ 
tive  work  analysis,  cognitive  modeling, 
cognitive  field  studies,  and  cognitive 
systems  engineering  (CSE)  as  relevant 
to  military  aviation  domains.  From  early 
work  involving  the  design  of  intelligent 
pilot-vehicle  interfaces  to  current  appli¬ 
cations  supporting  collaborative  activity 
in  airborne  warning  and  control  sys¬ 
tems  (AWACS)  operations,  there  is 
necessity  to  reinvent  terms  of  engage¬ 
ment  (see  Table  1). 

As  part  of  this  review/reinvention,  a 
general  framework  (see  Figure  1)  high¬ 
lights  different  emphasis  areas  taken  by 
CSE  practitioners.  The  framework  con¬ 
veys  issues,  models,  methods,  tools, 
application  examples,  and  provides 
broad  exposure  to  the  question,  “What 
is  the  use  of  cognitive  systems  engi¬ 
neering?”  Using  personal  perspectives, 
developments,  and  case  studies  (as  well 
as  other  practitioner  approaches  in  CSE] 
a  number  of  requirements,  trends,  and 
directions  are  discussed. 

The  categorical  structure  of  the  forth¬ 
coming  report  chapter,  designed  to 
address  this  basic-level  question,  is  com- 


Figure  1.  A  general  viewpoint  of  cognitive  systems 
engineering. 

posed  of  a  set  of  integrated  queries  (addressed  across 
several  personal  stages  of  development  that  represent 
an  order  of  emerging  maturity). 

As  differing  stages  of  personal  development  (in 
CSE)  are  explained  in  terms  of  challenges,  themes, 
contributions,  and  streams  of  influences,  significant 
advances  are  surveyed.  The  historical  threads  of 
ecological  psychology,  human  factors  engineering, 
and  knowledge  engineering  are  woven  together  to 
form  a  nexus  from  which  these  advances  emerge. 

To  complement  the  historical  significance  of 
advances  and  integrate  answers  to  the  queries 
asked,  a  “living  laboratory”  approach  (McNeese, 
1996)  is  developed  to  enable  CSE  practitioners  to 
become  a  community  of  learners.  The  living  labo¬ 
ratory  places  value  on  discovery  through  different 
venues,  concurrency,  ecological  validity,  feedback, 
mutually  informative  processes,  technological 
intervention,  and  the  willingness  to  broadly 
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Figure  2.  Conceptual  basis  for  the  Computer-Aided  Cognitive  Systems  Engineering  (CACSE)  (Potter,  Roth,  Woods,  &  Elm,  1998). 


approach  complex  problems  without  dogmatic, 
doctrinaire  biases. 

Two  recent  manifestations  of  the  living  lab  view 
are  the  Computer-Assisted  Cognitive  Systems 
Engineering  (CACSE)  framework/toolkit  (Potter, 
Roth,  Woods,  &  Elm,  1998)  and  a  new  systematic 
approach  for  conducting  cognitive  field  studies 
(McNeese,  Bautsch,  &  Narayanan,  1999).  CACSE 
will  be  briefly  described  here. 

The  CACSE  tool  integrates  CSE  and  software  engi¬ 
neering  analysis  for  the  purpose  of  creating  innova¬ 
tive  designs.  Figure  2  shows  the  conceptual  founda¬ 
tions  underlying  CACSE  development.  Tools  such  as 
CACSE  can  shape  where  CSE  will  lead.  When  cou¬ 
pled  with  strategies  for  cognitive  field  studies,  foun¬ 
dations  for  knowledge  as  design  are  implemented. 

Upon  review  of  15  years  of  work,  research,  and 
practice  in  CSE,  the  chapter  concludes  with  a  set 
of  lessons  learned  and  specific  challenge  points  for 
practitioners  that  are  posed  to  feed  new  trajecto¬ 
ries  for  discovery.* 
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Balancing  Practice-Centered 
Research  and  Design 


David  Woods  and 
Klaus  Christoffersen 


The  intersection  of  people,  tech¬ 
nology,  and  work  is  of  interest 
to  many  involved  with  research 
and  development.  Developers  and 
technologists  make  claims  about  how 
a  prospective  new  capability  or  new 
system  development  project  will 
impact  on  performance.  Sponsors  are 
caught  up  in  the  sweeping  dreams  per¬ 
mitted  by  technology  unfettered  from 
harsh  contexts  of  use,  yet  they  fear 
software  development  projects  that  fail 
to  provide  useful  tools  or  that  create 
unanticipated  negative  effects. 
Practitioners  and  observers  of  practi¬ 
tioners  at  work  note  repeated  forms  of 
clumsiness  in  the  technology  deployed 
and  unanticipated  side  effects  of 
change.  Researchers,  blinded  by  the 
glare  of  disciplinary  labels,  drastically 
reduce  situations  to  fit  into  a  laborato¬ 
ry  one  variable  at  a  time  yet  claim  pri¬ 
ority  in  the  search  for  generic  regular¬ 
ities.  Human  factors  practitioners  and 
usability  engineers  are  called  in  too 
late  to  repair  the  connection  between 
systems  and  use.  Research  results 
seem  irrelevant  to  design.  Design 
seems  local  and  unique. 

Research  and  development  at  the 
intersection  of  people,  technology,  and 
work  is  a  world  divided  and  hobbled. 
Innovation  is  tantalizing  yet  elusive.  In 
the  rush,  we  achieve  only  a  cumber¬ 
some  process  of  trial  and  error  (publi¬ 
cizing  the  extent  of  design  errors  and 
failures  would  be  bad  for  investment) . 
The  standard  metaphor  and  organiza¬ 
tional  construct  of  the  pipeline  has 
failed  given  the  possibilities  for  change 
and  the  predilection  for  new  technolo¬ 
gy  to  demand  connections  across  disci¬ 
plinary  boundaries.  Research  and 
development  in  this  area  is  a  world  too 
often  without  effective  interconnec¬ 
tions  and  cross-stimulation. 


We  provide  an  alternative  model  at  two  levels. 
The  first  attribute  is  complementarity  as  a  strategy 
for  practice-centered  research  and  design.  This  is 
the  foundational  strategy  behind  the  label  “cogni¬ 
tive  systems  engineering”  (and  related  labels  like 
“distributed  cognition”  and  “naturalistic  decision 
making”)  that  makes  it  a  substantive  alternative  to 
traditional  disciplinary  approaches.  In  other 
words,  all  of  the  new  labels  about  the  syntheses 
required  to  study  and  shape  the  intersection  of 
people,  technology,  and  work  are  only  superficial 
exercises  in  career  enhancement  unless  they  pro¬ 
vide  substance  to  complementarity.  Second,  the 
model  replaces  the  shopworn  cliche  of  a  research 
and  development  pipeline  (a  metaphor  that  may 
never  have  had  substance)  with  synchronization 
of  multiple,  parallel  cycles  of  learning  and  devel¬ 
opment  that  operate  at  different  time  scales. 
Interlocking  these  cycles  is  a  difficult  challenge 
(see  Figure  1). 

Researchers  are  connected  to  the  systems 
development  process  by  observing  and  abstract¬ 
ing  patterns  about  the  interaction  of  people, 
technology,  and  work.  These  researchers  must  be 
able  to  contribute  concepts  and  techniques  about 
what  would  be  useful  in  the  initial  stages  of  a 
development  cycle,  but  they  are  relaxed  from  the 
limited  resource  and  time  horizons  that  pressure 
development  of  real  working  systems. 

Those  working  to  advance  technology  for 
human  interaction  are  connected  to  the  systems 
development  process  by  using  or  participating  in 
studies  of  the  actual  effects  of  technology 
change.  Rather  than  just  measure  success  in 
terms  of  autonomous  machine  capabilities,  they 
can  look  to  the  research  base  for  empirically 
based  patterns  and  models  about  how  technolo¬ 
gy  developments  support  effective  collaboration 
with  human  practitioners. 

Effective  innovation  in  system  development 
depends  on  having  technological  advances  to 
draw  on  and  on  having  concepts  about  what 
may  be  useful  to  support  human  performance 
available  early  in  a  development  cycle  to  be 
able  to  identify  leverage  points  and  to  anticipate 
side  effects  of  change.  In  effect,  the  balancing 
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Figure  1.  The  engine  of  innovation:  Interlocking  cycles  of  research  and  development. 


act  needs  mechanisms  to  support  distributed  David  D.  Woods,  PhD,  and  Klaus 
innovation.  This  is  an  example  of  the  area  of  Christoffersen  are  with  the  Institute  for 
human-machine  systems  called  “computer-sup-  Ergonomics,  The  Ohio  State  University, 
ported  collaborative  work.”  Usually  this  work  is  Columbus,  Ohio. 
directed  at  practitioners,  designers,  or  man¬ 
agers.  Here  the  need  is  to  use  principles  for  col¬ 
laborative  work  and  the  technology  infrastruc¬ 
ture  for  connectivity  to  support  distributed 
innovation.  Doing  this,  as  is  building  any  kind 
of  collaboration,  requires  energy  and  invest¬ 
ment  in  coordinated  activities  across  the  multi¬ 
ple  parallel  cycles.* 
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Bridging  the  Gap  Between 
Cognitive  Analysis  and 
Cognitive  Engineering 


Emilie  M.  Roth,  James  Gualtieri, 
James  Easter,  Scott  S.  Potter,  and 
William  C.  Elm 


There  has  been  growing  interest 
in  using  cognitive  task  analysis 
(CTA)  to  understand  the 
requirements  of  cognitive  work 
(Schraagen,  Chipman,  &  Shalin,  2000; 
Vicente,  1999).  While  CTA  techniques 
have  proved  successful  in  illuminating 
the  sources  of  cognitive  complexity  in 
a  domain,  the  results  of  the  CTA  are 
often  only  weakly  coupled  to  the 
design  of  support  systems  [Potter, 
Roth,  Woods,  &  Elm,  2000).  A  critical 
gap  occurs  at  the  transition  from  CTA 
analysis  to  system  design,  where 
insights  gained  from  the  CTA  must  be 
translated  into  design  requirements. 

In  this  paper  we  describe  an  approach 
to  bridging  this  gap  and  present  a  visu¬ 
alization  that  we  developed  for  a  mili¬ 
tary  command  application  as  a  concrete 
illustration  of  the  approach. 

Our  approach  is  predicated  on  the 
premise  that  the  design  of  advanced 
visualizations  and  decision-aids  must 
be  grounded  in  an  understanding  of  the 
domain  of  practice  and  the  demands  it 
imposes  on  domain  practitioners. 
Design  activities  include: 

•  Capturing  the  critical  domain  rela¬ 
tionships  that  define  the  problem- 
space  that  the  domain  practition¬ 
ers  confront; 

•  Identifying  the  cognitive  tasks  and 
critical  decisions  that  arise  in  the 
domain  and  require  support; 

•  Identifying  the  information 
requirements  for  these  cognitive 
tasks  and  decisions; 

•  Defining  the  relationships  between 
decision  requirements  and  associ¬ 
ated  information  requirements  and 
user  interface  design  concepts; 

•  Exploring  techniques  to  implement 
these  design  concepts  into  powerful 
visualizations  of  domain  semantics. 


Figure  1  (see  page  15)  provides  a  visual  depic¬ 
tion  of  the  sequence  of  design  activities  and  asso¬ 
ciated  design  artifacts.  These  design  artifacts  cre¬ 
ate  a  continuous  design  thread  that  provides  a 
principled,  traceable  link  from  the  demands  of  the 
domain  as  revealed  by  the  CTA,  to  the  cognitive 
and  collaborative  processes  that  require  support, 
through  the  elements  of  the  decision  aid  that 
explicitly  address  those  support  requirements. 

We  recently  created  an  innovative  visualization 
to  support  military  commanders  in  choosing  com¬ 
bat  power  to  achieve  mission  objectives  using  this 
design  approach.  The  display  was  developed  as 
part  of  the  DARPA  Command  Post  of  the  Future 
program  (Logica  Carnegie  Group,  2000),  and  pro¬ 
vides  a  concrete  illustration  of  how  intermediate 
design  artifacts  can  be  used  to  provide  a  princi¬ 
pled,  traceable  link  from  cognitive  analysis  to 
design.  The  “choose  combat  power”  display  repre¬ 
sents  one  of  a  growing  number  of  examples  of  suc¬ 
cessful  systems  that  have  been  developed  using  a 
domain  analysis  approach  (e.g.,  Roth,  Lin,  Kerch, 
Kenney  &  Sugibayashi,  in  press;  Potter,  Roth, 
Woods  &  Elm,  2000). ■ 
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Design  Concepts 


Define  Relationship  Between  Decision 
Requirements  and  visualization  Concept 


Identify  Information  Requirements 
Identify  Cog  n id ve  Tasks 
Model  Critical  Domain  Relationships 


Figure  1.  A  sequence  of  linked  design  artifacts  is  used  to  create  a  continuous  design  thread  that  starts  with  a  representation  of 
domain  relationships  through  development  of  decision  support  requirements  to  creation  of  visualization  and  aiding  concepts  and  rapid 
prototypes  with  which  to  explore  the  design  concepts. 
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Cognitive  Work  Analysis  for 

Air  Defense  Applications  in  Australia 

Neelam  Naikar,  Gavan  Lintern,  and 
Penelope  Sanderson 


Cognitive  work  analysis  (CWA)  is 
one  of  the  possible  tools  that  a 
cognitive  systems  engineer  can 
use  for  analysing,  modeling,  designing, 
and  evaluating  complex  systems.  The 
specific  focus  of  CWA  is  on  supporting 
worker  adaptation  and  flexibility  in 
complex,  real-time  work  domains,  espe¬ 
cially  during  unanticipated  situations. 
CWA  is  therefore  particularly  well  suit¬ 
ed  to  military  domains  where  opposing 
forces  deliberately  create  unexpected 
situations  to  gain  a  tactical  advantage. 
With  this  in  mind,  Australia’s  Defence 
Science  and  Technology  Organisation 
(DSTO)  has  used  CWA  in  a  variety  of 
military  aviation  contexts. 

A  complete  CWA  typically  consists  of 
five  phases  (see  Figure  1  on  page  1 7) : 

•  Work  domain  analysis  which  focus¬ 
es  on  the  functional  purposes,  pri¬ 
orities  and  values,  purpose-related 
functions,  physical  functions,  and 
the  physical  objects  of  a  system; 

•  Control  task  analysis  which  focus¬ 
es  on  the  activity  that  must  occur 
in  the  work  domain  for  a  system  to 
achieve  its  functions  and  purposes; 
•  Strategies  analysis  which  focuses 
on  ways  the  tasks  or  activities  may 
be  carried  out; 

•  Socio-organizational  analysis 
which  focuses  on  who  carries  out 
the  work  and  how  it  is  shared;  and 
•  Worker  competencies  analysis  that 
focuses  on  the  knowledge,  train¬ 
ing,  capabilities,  and  expertise  that 
workers  need  to  carry  out  the 
work  of  the  system. 

All  of  the  phases  of  CWA  are  potential¬ 
ly  useful  throughout  a  system’s  life  cycle, 
although  some  phases  of  CWA  may  be 
more  useful  than  others  at  different 
points  in  the  life  cycle.  For  example,  dur¬ 


ing  the  initial  stages  of  a  system’s  life  cycle  work, 
domain  analysis  is  useful  for  defining  requirements 
because  it  identifies  why  a  new  system  is  needed, 
what  its  environmental  context  will  be,  and  what 
functions  must  be  implemented.  Flowever,  more 
detail  about  the  activity  of  the  system  is  needed  dur¬ 
ing  the  specifications  stage  at  which  point  a  control 
task  analysis  becomes  an  important  contributor. 
During  the  system  design  stage,  all  components  of  a 
CWA  are  typically  expected  to  contribute.  The  vari¬ 
ous  phases  of  CWA  can  also  be  used  for  evaluating 
competing  designs,  defining  training  requirements, 
and  considering  potential  upgrades.  Finally,  CWA 
can  also  help  to  support  a  decision  to  retire  a  system 
when  a  work  domain  analysis  of  the  broader  work 
context  shows  that  the  current  system  is  no  longer 
needed  or  competitive. 

Research  at  DSTO  has  demonstrated  several  of  the 
potential  benefits  of  CWA.  One  of  the  most  success¬ 
ful  applications  of  CWA  was  to  Australia’s  acquisi¬ 
tion  of  a  fleet  of  airborne  early  warning  and  control 
(AEW&C)  aircraft.  A  work  domain  analysis  of 
AEW&C  was  used  to  evaluate  competing  designs 
from  three  potential  manufacturers.  This  approach 
fostered  an  integrated  evaluation  of  the  three 
designs  by  the  AEW&C  Project  Office.  It  also 
allowed  the  Project  Office  to  express  the  results  of 
the  evaluation  in  terms  of  military  utility  as  opposed 
to  technical  properties.  Moreover,  the  work  domain 
analysis  served  as  a  “sanity  check”  because  it  sup¬ 
ported  an  explicit  evaluation  of  the  impact  of  phys¬ 
ical  design  properties  on  higher-level  functionality. 
CWA  has  also  contributed  to  the  analysis  of  AEW&C 
activities  and  the  evaluation  of  AEW&C  human-sys¬ 
tem  integration  and  automation.  In  addition,  DSTO 
researchers  have  also  been  successful  in  using  CWA 
to  define  training  needs  in  support  of  the  Australian 
Defence  Force’s  acquisition  of  a  training  system  for 
the  F/A-18  fighter  aircraft. 

In  conclusion,  within  the  context  of  military  avi¬ 
ation,  DSTO  has  found  that  CWA  has  provided 
valuable  insights  concerning  matters  such  as  inter¬ 
face  design,  training  program  specification, 
research  program  design,  intelligent  agent  model¬ 
ing,  and  so  forth.  The  products  of  CWA  have  also 
been  found  to  be  reusable  for  different  purposes 
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Figure  1.  Five  phases  of  CWA  with  iconic  representations  of  their  most  familiar  analytic  products 


over  a  system’s  life  cycle.  Furthermore,  CWA  has 
become  an  intellectual  framework  at  DSTO  within 
which  human  factors  practitioners,  training  spe¬ 
cialists,  simulator  builders,  cognitive  scientists, 
and  operations  researchers  can  communicate  and 
dovetail  their  activities.* 
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Analyzing  the  Cognitive  System 
From  a  Perceptual  Control  Theory 
Point  of  View 

Keith  C.  Hendy,  David  Beevis, 
Frederick  Lichacz,  and 
Jack  L.  Edwards 


A  closed  loop,  negative  loop  gain, 
feedback  system  is  an  error-cor¬ 
recting  system.  The  inverse  of 
this  proposition  is  that  all  error-correct¬ 
ing  systems  can  be  reduced  to  a  closed 
loop,  negative  gain,  feedback  system.  If 
these  propositions  are  true,  and  the 
human  is  seen  as  exhibiting  error-cor¬ 
recting  behavior,  then  William  T. 
Powers’  claim  that  all  human  behavior 
occurs  as  a  result  of  a  perceptually  driv¬ 
en,  goal  referenced,  feedback  system 
(Powers,  1973),  should  come  as  no  sur¬ 
prise.  This  is  the  tenet  of  Perceptual 
Control  Theory  (PCT),  depicted  in 
Figure  1 .  At  best  PCT  provides  a  truthful 
explanation  of  how  humans  form  and 
emit  behaviors;  at  the  very  worst  PCT  is 
a  normative  model  of  human  behavior. 


Figure  1:  The  Perceptual  Control  Theory 
(PCT)  model. 

The  PCT  model  suggests  a  multilay¬ 
ered  system,  with  multiple  goals  provid¬ 
ing  the  reference  points  for  a  hierarchi¬ 
cal  organization  of  control  loops.  These 
loops  provide  control  at  many  levels — 


from  the  lowest  levels  of  sensory  processing, 
upward  to  the  satisfaction  of  abstract  goals  such  as 
the  need  for  self-esteem  and  actualization.  In  PCT 
terms,  an  emitted  action  or  behavior  is  in  response 
to  the  presence  of  an  error,  or  difference,  signal. 
The  emitted  action  is  transmitted  purposefully, 
with  the  intention  of  changing  the  state  of  the 
world  so  that  the  operator’s  perception  can  be 
made  to  match  a  desired  state  or  goal.  This 
reduces  the  error  signal  to  zero.  The  constraints  on 
human  information  processing,  within  the  mod¬ 
ules  of  the  PCT  loop,  are  described  by  the 
Information  Processing  (IP)  model  (Hendy,  Farrell, 
&  East,  2000;  Hendy,  Liao,  &  Milgram,  1997). 
Together  the  IP/PCT  models  provide  a  strong  inte¬ 
grating  framework  for  analyzing  and  predicting 
human  information  processing  behaviors  (Hendy 
&  Farrell,  1997). 

The  starting  point  for  the  design  of  any  complex 
system  should  be  analysis.  For  systems  where 
human  functions  are  predominantly  “cognitive,” 
the  method  of  analysis  should  capture  this  essen¬ 
tially  human  activity.  Traditionally,  human  engi¬ 
neering  analyses  have  been  based  on  a  hierarchi¬ 
cal  decomposition  of  system  missions,  functions, 
and  tasks  (MFTA).  Perceptual  Control  Theory, 
together  with  the  IP  model,  provides  a  theoretical 
framework  for  guiding  this  process.  PCT  reorien¬ 
tates  the  approach  from  a  serial  process  of  function 
analysis,  function  allocation,  task  analysis  process, 
to  an  integrated  process  of  a  hierarchically  direct¬ 
ed  goal  analysis  (or  HGA).  PCT-HGA  combines 
the  previously  separate  processes  into  a  single  uni¬ 
fied  process.  With  PCT  it  is  inescapable  that  goals 
at  all  levels  are  candidates  for  assignment  to  an 
agent  (human  or  machine).  This  is  a  major  point 
of  departure  between  traditional  MFTA  and  the 
IP/PCT-based  method. 

Two  new  analyses  emerge  from  the  PCT  frame¬ 
work.  The  first,  a  stability  analysis,  looks  to  see  if 
certain  external  variables  can  be  simultaneously 
under  multiple  control.  If  conflicting  goals  or 
incompatible  internal  perceptual,  cognitive,  or 
machine  functions  could  cause  these  multiple  con¬ 
trol  situations  to  be  unstable,  then  the  designer  has 
to  find  a  way  to  separate  control  or  otherwise 
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ensure  stability.  The  second  analysis  looks  at  the 
upward  flow  of  information  in  the  system.  Each 
goal  is  examined  to  see  how  information  existing 
at  the  subgoal  level  flows  up  to  the  level  above. 
Both  analyses  potentially  identify  new  goals  that 
must  be  accommodated  by  interface  design. 

A  PCT-based  HGA  was  performed  on  a  single 
segment  of  a  land  forces  command  and  control 
environment  that  had  previously  been  analyzed  by 
traditional  MFTA.  The  PCT-based  HGA  identified 
37  additional  activities  that  had  been  missed  in  the 
original  analysis.  The  success  of  the  PCT  analysis 
in  identifying  additional  goals,  missed  in  the  tradi¬ 
tional  MFTA,  is  heartening.  The  new  goals  were  all 
associated  with  support  to  higher  level  intent.  For 
this  particular  application,  it  would  seem  that 
established  command  and  control  procedures  had 
largely  addressed  the  issue  of  stability,  as  the 
potential  for  simultaneous  control  was  not  a  factor 
in  the  analysis  of  this  segment  of  operation.* 
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Cognitive  Cockpit  Engineering: 

Coupling  Functional  State  Assessment, 

Task  Knowledge  Management  and 
Decision  Support  for 
Context  Sensitive  Aiding 

Robert  M.  Taylor,  Michael  C.  Bonner,  Blair  Dickson,  Howard  Howells,  Christopher  A.  Miller, 
Nicholas  Milton,  Kit  Pleydell-Pearce,  Nigel  Shadbolt,  Jeni  Tennison,  and  Sharon  Whitecross 


Future  offensive  air  systems  will  be 
highly  automated,  but  human 
involvement  will  still  be  needed. 
In  complex,  rapidly  changing  military 
environments  maintaining  effective 
human  cognitive  involvement  is  a  sig¬ 
nificant  challenge.  A  human-centered 
approach  to  system  design  is  needed 
that  is  based  on  human  cognitive 
requirements.  Pilot  judgment  provides 
context  sensitivity  that  is  hard  to  auto¬ 
mate.  In  air  systems  involving  high  lev¬ 
els  of  automation,  rather  than  replacing 
the  pilot,  technology  is  needed  to  assist 
the  future  aircrew  in  cognitive  work.  To 
be  responsive  to  changing  mission 
requirements,  in  particular  for  in-flight 
situation  assessment  and  mission  re¬ 
planning,  adaptive  and  context-sensi¬ 
tive  support  will  be  needed.  To  be 
responsive  to  the  changing  require¬ 
ments  of  the  human  operator,  technolo¬ 
gy  also  needs  to  be  influenced  by  the 
individual’s  physiological  and  behav¬ 
ioral  state.  This  would  adaptively 
respond  to  indications  of  overload,  dis¬ 
traction  or  performance  difficulty,  and 
to  the  possibility  of  incapacitation  in 
flight,  providing  a  pilot  safety  net. 

Recognizing  this  challenge,  the  UK 
Ministry  of  Defence  (MOD)  in  conjunc¬ 
tion  with  the  Defence  Evaluation  and 
Research  Agency  (DERA)  are  conducting 
a  three-year  program  of  applied  research 
in  cognitive  engineering  looking  at  intel¬ 
ligent  knowledge-based  decision  aiding 
for  military  fast  jet  pilots.  The  Cognitive 
Cockpit  (COGPIT)  project  (see  Figure  1) 
seeks  to  couple  cognitive  technologies 
for  pilot  functional  state  assessment  and 


knowledge-based  systems  for  situation  assessment 
and  decision  support  with  concepts  and  technolo¬ 
gies  for  adaptive  automation  and  cockpit  adaptive 
interfaces. 

Pilot  functional  state  monitoring  is  in  its  infancy 
in  providing  on-line  measurement  for  task  adapta¬ 
tion.  In  contrast,  task  knowledge  management  and 
decision  support  for  context-sensitive  aiding 
involves  the  application  of  relatively  mature  tech¬ 
nology.  Coupling  these  should  provide  a  system 
capable  of  recognizing  the  need  for  automation  to 
achieve  a  mission  objective  and  of  providing 
instructions  to  the  operator  on  how  to  achieve  it, 
and/or  implement  the  required  automation.  Initial 
development  work  indicates  that  on-line  pilot 
functional  state  assessment  is  feasible  with  current 
computing  power  and  could  provide  useful  infor¬ 
mation  for  cockpit  and  task  adaptation.  In  particu¬ 
lar,  the  increased  power  of  individual  profiles  for 
developing  customized  adaptations  seems  a  highly 
promising  development. 

The  work  shows  how  a  knowledge  engineering 
methodology  can  provide  useful  on-line  knowl¬ 
edge-based  systems  (KBS)  support  for  pilot  replan¬ 
ning  tasks,  with  potential  for  wider  application. 
The  traditional  knowledge  acquisition  bottleneck 
has  been  significantly  reduced  by  the  provision  of 
a  structured  methodology  and  tool  set. 
Demonstration  has  highlighted  the  criticality  of  the 
timing  of  KBS  advice  in  context.  The  work  has  also 
shown  how  useful  assistance  in  the  management  of 
cockpit  interfaces,  tasks,  and  automation  can  be 
provided  by  a  tasking  interface  system  based  on  a 
shared  task  model.  The  development  of  an  effective 
tasking  interface  manager  with  which  pilots  can 
interact  easily,  is  critical.  Although  it  is  relatively 
easy  to  track  tasks  instantiated  in  a  mission  plan,  it 
becomes  very  difficult  to  track  tasks  that  deviate 
from  the  plan.  Tracking  deviations  requires  the  sys¬ 
tem  to  infer  pilot  intent,  which  is  problematic. 
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Figure  1.  Initial  conceptual  prototype  for  the  DERA 
Cognitive  Cockpit. 

Current  cockpit  work  seeks  to  identify  the  pre¬ 
cise  methods  for  cockpit  adaptation  and  their  ben¬ 
efits  and  to  determine  the  optimization  of  con¬ 
trol/display  interfaces,  in  particular  for  voice  dia¬ 
logue,  and  the  helmet-mounted  display  and  3D 
audio  ramifications.  A  key  step  has  been  the 
design  of  a  use-centric  system  for  cognitive  control 
of  levels  of  adaptive  automation  that  avoids 
unwanted  automation  surprises. 

The  COGPIT  project  has  responded  to  the  chal¬ 
lenge  of  increasing  automation  in  a  complex  envi¬ 
ronment  where  human  involvement  is  needed  in 
critical  decisions.  In  seeking  to  couple  cognitive 


technologies  for  pilot  aiding  with  exten¬ 
sive  use  of  cognitive  engineering  meth¬ 
ods,  the  project  provides  a  particularly 
human  centric  approach  to  use-centric 
design.  The  benefits  of  real-time  decision 
support,  and  on-line  cockpit  task  adapta¬ 
tion,  seem  likely  to  become  evident  and 
realizable  in  the  near  future.  But,  it 
seems  that  further  work  will  be  needed 
to  provide  the  benefits  of  real-time  adap¬ 
tation  to  cognitive  functional  state.* 
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Beyond  Cognitive  Engineering: 

Assessing  User  Affect  and  Belief  States  to 
Implement  Adaptive  Pilot-Vehicle  Interaction 

Eva  Hudlicka  and 
Michael  D.  McNeese 


The  mutual  influence  of  cognitive 
schemata  and  contextual  con¬ 
straints  is  considered  the  accept¬ 
ed  basis  for  cognitive  system  engineer¬ 
ing  practices  (Rasmussen,  Pejtersen,  & 
Goodstein,  1994).  Less  generally  accept¬ 
ed  is  the  fact  that  affective  states  can 
also  dramatically  influence  human  per¬ 
formance  and  decision  making  via 
effects  on  attention,  perception,  situa¬ 
tion  assessment,  and  ultimately  action 
selection  (LeDoux,  1992;  Williams, 
Watts,  MacLeod,  &  Mathews,  1997). 

Currently  few  human-machine  sys¬ 
tems  attempt  to  dynamically  assess  and 
adapt  to  the  users’  affective  and  belief 
states.  This  shortcoming  can  lead  to 


nonoptimal  behavior  at  best  and  critical  errors 
with  disastrous  consequences  at  worst.  This  is 
increasingly  evidenced  by  a  variety  of  accidents 
and  incidents  broadly  attributed  to  “human  error” 
that  exist  in  a  number  of  settings,  including  mili¬ 
tary  aviation,  where  conditions  of  heightened 
stress  are  common. 

To  address  these  challenges,  we  developed  an 
Affect  and  Belief  Adaptive  Interface  System  (ABAIS) . 
ABAIS  uses  a  methodology  designed  to  adapt  the 
human-machine  interface  to  the  user’s  affective  state 
and  situation-specific  beliefs  that  might  influence 
performance  (Hudlicka  &  Billingsley,  1999).  The 
methodology  consists  of  four  steps:  sensing/infer¬ 
ring  user’s  affective  state  and  performance-relevant 
beliefs;  identifying  their  potential  effects  on  per¬ 
formance;  selecting  a  compensatory  strategy;  and 
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Figure  1.  ABAIS  architecture  implementing  the  affect-belief  adaptive  methodology. 
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implementing  this  strategy  in  terms  of  specific 
graphic  user  interface  (GUI)  adaptations. 

The  ABAIS  prototype  implements  this  methodol¬ 
ogy  in  terms  of  an  architecture  shown  in  Figure  1. 
The  architecture  consists  of  four  modules:  User 
State  Assessment  provides  a  framework  for  inte¬ 
grating  a  variety  of  methods  to  identify  the  user’s 
affective  and  belief  state  (e.g.,  knowledge-based, 
self-reports,  diagnostic  tasks,  physiological  sens¬ 
ing).  Impact  Prediction  integrates  generic  empiri¬ 
cal  findings  with  the  results  of  task-specific 
Cognitive  Affective  Personality  Task  Analysis 
(CAPTA)  (Hudlicka,  2000)  to  predict  the  most  like¬ 
ly  effects  of  user  states  on  performance.  Strategy 
Selection  combines  the  CAPTA  results  with  indi¬ 
vidual  preferences  to  derive  an  appropriate  com¬ 
pensation  strategy.  Finally,  GUI  Adaptation  imple¬ 
ments  this  strategy  by  modifying  the  content 
and/or  format  of  the  user  interface  (see  Figure  2). 

The  ABAIS  prototype  was  developed  and 
demonstrated  in  the  context  of  an  Air  Force  com¬ 
bat  task  simulation.  ABAIS  assessed  the  pilot’s 
anxiety  and  belief  states  via  a  knowledge-based 
approach  using  information  from  a  variety  of 
sources  (e.g.,  task  characteristics,  pilot  personality, 
etc.),  predicted  the  effects  of  user  state  on  per¬ 
formance,  and  suggested  and  implemented  specif¬ 
ic  GUI  adaptation  strategies  based  on  the  pilot’s 
information  presentation  preferences  (e.g.,  modi¬ 
fied  icon/display  to  capture  attention,  etc.) . 

Preliminary  results  indicate  feasibility  of  the 
ABAIS  approach,  raise  a  number  of  further  research 
questions,  and  suggest  specific  requirements  for  a 
successful,  operational  affect  and  belief  adaptive 
interface  (e.g.,  limiting  the  number,  type,  and  reso¬ 
lution  of  affective  and  belief  states;  using  multiple 
methods  and  individualized  data  for  user  state 
assessment;  implementing  “benign”  adaptations— 
adaptations  should  never  limit  access  to  existing 
information) . 

This  work  represents  an  attempt  to  move  beyond 
the  traditional  cognitive  and  psychophysical  factors 
in  designing  human-machine  interfaces  by  explicit¬ 
ly  integrating  affective  and  personality  considera¬ 
tions  into  the  design  process  via  CAPTA.  While  the 
initial  prototype  was  developed  within  a  military 
aviation  task  context,  we  believe  that  the  results  are 
applicable  to  a  broad  variety  of  nonaviation  and 
nonmilitary  domains.* 
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Figure  2.  Summary  of  GUI  adaptations. 
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